feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Claude) by DevelopmentCats · Pull Request #10 · coder/coder-skill-scanner

DevelopmentCats · 2026-06-22T19:55:50Z

What this does

Flips the scheduled scan from --no-llm to SkillSpector's two-stage analyser (static rules + LLM semantic pass).

Provider is anthropic against api.anthropic.com directly, model claude-sonnet-4-6 (SkillSpector's bundled meta_analyzer slot default; 1M context window; SkillSpector's model_registry.yaml already knows its token limits so no runtime fallback warning). SkillSpector cannot be routed through Coder's AI Gateway today, so the Anthropic API key is on a separate billing line from Coder usage — the four-bullet reason chain is in docs/CALIBRATION.md under "Provider choice and the workflow gap." Short version: SkillSpector pipes every provider through langchain_openai.ChatOpenAI (OpenAI /v1/chat/completions shape), aibridge only exposes that path under /openai, and SkillSpector's anthropic provider hardcodes https://api.anthropic.com/v1/ and ignores ANTHROPIC_BASE_URL. Empirically verified end-to-end against the workspace's aibridge.

Stack

Stacked on PR #1. Merge that first, then re-target this to main and merge.

Measured impact on the five in-tree skills

Measured by running skillspector scan twice on each upstream skill (once with --no-llm, once with LLM mode on, model gpt-4.1-mini through aibridge during development). Methodology and the model-swap caveat are in docs/CALIBRATION.md.

Skill	static score	static verdict	LLM score	LLM verdict	findings Δ
coder/coder-modules	10	clean	0	clean	1 → 0 (-1)
coder/coder-templates	10	clean	0	clean	1 → 0 (-1)
coder/modules	0	clean	0	clean	0 → 0
coder/setup	100	malicious	26	clean	23 → 2 (-21)
coder/templates	0	clean	0	clean	0 → 0
TOTAL					25 → 2 (-23)

coder/setup flips malicious → clean. The LLM filtered all 23 static-only findings as context-aware false positives (EA2 hits on safeguard prose, MP2 hits on PNG assets, SC2 hits on curl coder.com/install.sh, PE3 hits on the skill's own scratch files, etc.) and surfaced two new MEDIUM SQP-2 findings: the GitHub device-flow scripts write the OAuth token and session config to disk without an explicit notification. Those are real and minor; the cleanest fix is a one-line echo upstream in coder/skills, not a change here.

Model swap caveat: production runs against Claude Sonnet 4.6, not gpt-4.1-mini. The 25 → 2 delta above measures SkillSpector's LLM semantic pass as a capability; per-finding counts may shift one or two either way under Claude. The verdict-band outcomes are robust to that drift because every static finding on the four other in-tree skills is well below the suspicious_risk_score: 51 cutoff to begin with, so even a 100% no-filter LLM still leaves them clean. Recalibration against Claude lands in a follow-up PR once the secret is wired in and the first production scan runs.

Setup before merge

Add ANTHROPIC_API_KEY as a repo secret (from console.anthropic.com).
Apply the workflow diff to .github/workflows/scan.yaml (see collapsible below). The Coder Agents app on this repo doesn't have workflows: write, so the bot couldn't commit that file. Either grant the permission and ping me to re-push, or paste the diff yourself.
After merge: Actions → scan → Run workflow.

Graceful fallback

If the secret isn't set, the workflow emits a warning and runs --no-llm. A fresh fork keeps producing valid (lower-precision) scans without any setup.

Why scan.yaml isn't in this PR

GitHub Apps need an explicit workflows: write permission separate from contents: write to modify .github/workflows/*. The Coder Agents installation on this repo currently has contents: write but not workflows: write, so the agent could not commit the scan.yaml change. Two ways forward:

Quick path: paste the diff below into scan.yaml via the GitHub web editor. Takes about a minute.
Permanent path: grant the Coder Agents GitHub App workflows: write for this repo. After that I can re-push and this PR's workflow file will be included.

Either produces the same merged state.

Workflow file diff (to paste into .github/workflows/scan.yaml)

Three edits inside the scan job.

1. Add new step after Verify skill path exists:

      - name: Determine LLM mode
        id: llm
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          set -euo pipefail
          if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
            echo "extra_flags=" >> "$GITHUB_OUTPUT"
            echo "SkillSpector LLM mode: enabled (anthropic provider, api.anthropic.com)." >&2
          else
            echo "extra_flags=--no-llm" >> "$GITHUB_OUTPUT"
            echo "::warning::ANTHROPIC_API_KEY secret not set; SkillSpector will run with --no-llm. See README \"One-time setup\" to enable the LLM semantic pass."
          fi

2. Replace the SkillSpector (JSON) step:

      - name: SkillSpector (JSON)
        if: steps.path_check.outputs.drift == 'false'
        continue-on-error: true
        env:
          SKILLSPECTOR_PROVIDER: anthropic
          SKILLSPECTOR_MODEL: claude-sonnet-4-6
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mkdir -p out
          skillspector scan "source/${{ matrix.skill_path }}" \
            ${{ steps.llm.outputs.extra_flags }} \
            --format json \
            --output "out/skillspector.json" || true

3. Same swap for SkillSpector (SARIF):

      - name: SkillSpector (SARIF)
        if: steps.path_check.outputs.drift == 'false'
        continue-on-error: true
        env:
          SKILLSPECTOR_PROVIDER: anthropic
          SKILLSPECTOR_MODEL: claude-sonnet-4-6
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mkdir -p out
          skillspector scan "source/${{ matrix.skill_path }}" \
            ${{ steps.llm.outputs.extra_flags }} \
            --format sarif \
            --output "out/skillspector.sarif" || true

Decision log

Anthropic API direct over aibridge. Aibridge cannot route SkillSpector at Claude today: SkillSpector's anthropic provider hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL; SkillSpector pipes every provider through langchain_openai.ChatOpenAI (OpenAI /v1/chat/completions shape); aibridge does proxy Claude, but only in Anthropic's native /v1/messages shape under its /anthropic path, and does not mount /v1/chat/completions on that path. Verified empirically: POST $ANTHROPIC_BASE_URL/v1/messages returns Claude output; POST $ANTHROPIC_BASE_URL/v1/chat/completions returns route not supported. The trade-off (separate billing line vs Claude-class model) was made deliberately; using openai against aibridge with gpt-4.1-mini is a viable alternative documented in docs/CALIBRATION.md. The provider line flips back to openai without any workflow change if aibridge later adds the missing route or SkillSpector gains a native-Anthropic integration.
claude-sonnet-4-6. SkillSpector's bundled meta_analyzer slot default; the provider docstring describes that slot as "cheaper for the high-volume filter pass," which is precisely the per-finding intent classification this scanner does. Bumped from the earlier claude-sonnet-4-5-20250929 pick because 4-6 is one generation newer, widens context from 200K to 1M (reduces chunking on larger skills), and is in SkillSpector's model_registry.yaml (no runtime fallback warning). Bare alias is used because the dated suffixes for the 4-6 series return 404 from Anthropic's API as of 2026-06-23; the bare alias is the canonical ID. Override in config.yaml to claude-opus-4-6 (SkillSpector's DEFAULT_MODEL, ~5x cost) or to claude-opus-4-8 / claude-fable-5 if a higher tier is warranted later.
Calibration table was measured on gpt-4.1-mini. Captured during the aibridge-routing exploration, before the provider swap. Kept in docs/CALIBRATION.md because the LLM-pass capability — not the model choice — is what drives the 25 → 2 reduction, and the verdict-band outcomes are robust to model differences for these five inputs. A follow-up PR refreshes the numbers with Claude.
Graceful --no-llm fallback. A fresh fork should produce something useful immediately, not 404 the publish pipeline because the operator hasn't added the secret yet. The workflow warning makes the degraded state visible.
No CI change. ci.yaml uses inline pytest fixtures and never invokes skillspector live, so no inference cost on PR review.
No schema or verdict-math change. LLM mode shifts which findings reach the verdict, not how the verdict is computed. CALIBRATION.md walks through this explicitly.
Permissions manifest dropped from the near-term plan. Was originally proposed to fix coder/setup's false-positive avalanche. LLM mode handles that better (23 → 2 findings). The two remaining findings on coder/setup are real and can be fixed in a 4-line PR against coder/skills rather than justified via a manifest. The manifest design is captured for future use when we onboard third-party skills.

This PR was prepared with help from Coder Agents.

Document the new llm.provider config knob (default nv_build) and the workflow contract: empty flags + workflow appends --no-llm dynamically when the matching credential secret is unset. Removes --no-llm from the static flags list now that the workflow drives it. This commit was prepared with help from Coder Agents.

Document what flipping LLM mode on does (and does not do) to the verdict math, the precision delta we expect, and what to expect for the five in-tree skills. Adds "LLM provider changes" to the "When to revisit" list. This commit was prepared with help from Coder Agents.

Update step 3 of the architecture summary to reflect that the scheduled scan now runs SkillSpector with the LLM semantic pass on by default. Add a new "One-time setup on the repo" section that lists the three repo-level configurations needed for a useful scan, including the new LLM credential secret. Mirror the LLM secret note into "Forking for your own catalogue". This commit was prepared with help from Coder Agents.

Copilot

Pull request overview

Updates the scanner configuration and documentation to support running NVIDIA SkillSpector with an optional LLM semantic pass (with a fallback to static-only mode), as part of the scheduled scan pipeline described in this repo.

Changes:

Extend config.yaml with a scanners.skillspector.llm configuration block and make scanners.skillspector.flags empty by default.
Document the LLM semantic pass behavior, expected precision delta, and repo one-time setup steps in README.md and docs/CALIBRATION.md.
Shift responsibility for driving --no-llm to the scheduled workflow (though the workflow change itself is not included in this PR’s diff).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
README.md	Updates architecture and adds one-time setup guidance, including LLM credential setup.
docs/CALIBRATION.md	Adds an “LLM semantic pass” section explaining expected effects and limitations.
config.yaml	Adds `scanners.skillspector.llm` block and removes the default `--no-llm` flag from config-driven flags.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Swap the default LLM provider from nv_build (free NVIDIA Build) to anthropic with model pinned to claude-sonnet-4-6. Rationale: - Removes the second-vendor signup. The Coder org already has an Anthropic billing relationship, so the credential is one secret away from working. - Sonnet 4.6 is roughly 5x cheaper than the anthropic default (Opus 4.6) and is well matched to SkillSpector's LLM pass, which is finding-by-finding intent classification rather than long-form reasoning. Cost ballpark for 5 skills x 4 scans/day is small. - The other provider options (anthropic_proxy via Vertex, openai via any OpenAI-compatible gateway, nv_build) stay documented in the config comments and are still a one-line swap. This commit was prepared with help from Coder Agents.

Follow-up to the provider swap. The one-time-setup section now points at console.anthropic.com and ANTHROPIC_API_KEY instead of build.nvidia.com / NVIDIA_INFERENCE_KEY. This commit was prepared with help from Coder Agents.

Addresses copilot-pull-request-reviewer review on config.yaml line 46 and line 57: the previous comments described workflow behavior as implemented ("the workflow appends --no-llm dynamically") when in fact .github/workflows/scan.yaml still hardcodes --no-llm in this PR. The new wording describes the contract between this file and scan.yaml and explicitly notes that the matching workflow edit is committed separately because the Coder Agents GitHub App lacks `workflows: write`. Also swaps the provider from anthropic/claude-sonnet-4-6 to openai/gpt-4.1-mini. The anthropic provider hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL, so it cannot route through Coder's aibridge. The openai provider does, and gpt-4.1-mini was empirically validated against the five in-tree skills (results in CALIBRATION.md). This commit was prepared with help from Coder Agents.

Addresses copilot-pull-request-reviewer review on docs/CALIBRATION.md line 121: the LLM section described auto-enablement based on the credential secret, but scan.yaml in this branch still hardcodes --no-llm and that contract is not in effect yet. This pass: - Adds a measured table showing the five-skill before/after under LLM mode (was upstream's hand-wavy 70%-to-87% precision estimate). Catalogue-wide findings: 25 to 2; coder/setup: malicious to clean. - Adds an explicit "workflow gap" callout explaining that scan.yaml still hardcodes --no-llm in this branch and pointing readers at the PR description for the matching diff. - Documents the provider choice in plain language: anthropic provider cannot be steered at aibridge because it hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL; openai provider works and gpt-4.1-mini is the cost/quality sweet spot. - Drops the previous "bringing coder/setup below suspicious requires the permissions-manifest layer" framing. With LLM mode on, coder/setup is already clean. This commit was prepared with help from Coder Agents.

…aibridge Addresses copilot-pull-request-reviewer reviews on README.md (architecture step 3 at line 14, the one-time setup block, the graceful-fallback paragraph, and the forking step 5 at line 125): the text described workflow behavior that is not implemented in this branch because .github/workflows/scan.yaml still hardcodes --no-llm. Fixes: - Add a note block right after the architecture summary spelling out that step 3's LLM behavior requires the matching scan.yaml edit, and pointing readers at the PR description for the diff. - Update step 3 of the one-time-setup section to set the OpenAI provider's secret and add OPENAI_BASE_URL as a variable. Adds a second note block immediately after the setup list flagging the workflow-file dependency again at point-of-use. - Update forking instructions step 5 to point at the new setup section and call out that confirming scan.yaml exports the secret is required for LLM mode. - Swap the API-key reference from console.anthropic.com / ANTHROPIC_API_KEY to a Coder AI Gateway token / OPENAI_API_KEY. The anthropic provider in SkillSpector hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL, so it cannot be steered at aibridge; the openai provider works. This commit was prepared with help from Coder Agents.

SkillSpector's anthropic provider hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL, and aibridge does not mount /v1/chat/completions on its /anthropic path (only the native /v1/messages shape). Routing SkillSpector at Claude through aibridge is therefore not possible today without either patching SkillSpector or adding a new route to aibridge. Use the Anthropic API directly instead. This trades the aibridge billing-line consolidation for a Claude-class model; the choice and the trade-off are documented in docs/CALIBRATION.md.

…RL var Three-step one-time setup now: Pages, workflow permissions, and a single secret (ANTHROPIC_API_KEY from console.anthropic.com). The OPENAI_BASE_URL variable is no longer used since the provider is anthropic against api.anthropic.com directly. The workflow-file note spells out the matching workflow env block (SKILLSPECTOR_PROVIDER, SKILLSPECTOR_MODEL, ANTHROPIC_API_KEY) that has to land in scan.yaml.

… caveat The measured 25 -> 2 false-positive reduction was captured against gpt-4.1-mini through aibridge during development; production hits claude-sonnet-4-5 via the Anthropic API directly. Verdict-band outcomes are robust to the model swap because every non-coder/setup in-tree skill scores well below the 51 suspicious cutoff even without LLM filtering, but the per-finding counts may shift one or two either way. Recalibration follow-up planned once production data lands. Also captures the four reasons aibridge cannot route SkillSpector at Claude today.

…-4-6 sonnet-4-6 is the bundled meta_analyzer-slot default in SkillSpector's anthropic provider — the slot whose docstring explicitly describes it as 'cheaper for the high-volume filter pass,' which is exactly the per-finding intent classification this scanner does. It also widens the context window from 200K (Sonnet 4.5) to 1M, reducing chunking for larger skills, and SkillSpector's bundled model_registry.yaml already knows its token limits so there is no runtime fallback warning. Bare alias used rather than a dated suffix: probing Anthropic's /v1/models endpoint on 2026-06-23 confirmed every dated suffix for the 4-6 series (e.g. claude-sonnet-4-6-20251015) returns 404, while the bare alias returns 200. The bare alias is the canonical ID Anthropic accepts.

config.yaml: collapse the llm: block to provider + model + a one-line flags: comment. The aibridge backstory, model justification, provider options table, and "contract with scan.yaml" prose all live in docs/CALIBRATION.md and the PR description; reproducing them next to the two values that actually matter is noise. README.md and docs/CALIBRATION.md: drop the two "this PR is in a weird in-between state" callout blocks. The PR description already covers the workflow split; the docs do not need to carry it.

DevelopmentCats added 3 commits June 22, 2026 14:50

DevelopmentCats requested review from bpmct and phorcys420 as code owners June 22, 2026 19:55

Copilot AI review requested due to automatic review settings June 22, 2026 19:55

Copilot started reviewing on behalf of DevelopmentCats June 22, 2026 19:56 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Comment thread README.md Outdated

Comment thread docs/CALIBRATION.md Outdated

Comment thread config.yaml Outdated

Comment thread config.yaml Outdated

Comment thread README.md Outdated

Comment thread README.md Outdated

Comment thread README.md Outdated

DevelopmentCats added 2 commits June 22, 2026 17:23

DevelopmentCats changed the title ~~feat(scanner): enable SkillSpector LLM semantic pass~~ feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Sonnet) Jun 22, 2026

DevelopmentCats added 3 commits June 23, 2026 15:30

DevelopmentCats changed the title ~~feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Sonnet)~~ feat(scanner): enable SkillSpector LLM semantic pass via Coder aibridge Jun 23, 2026

DevelopmentCats added 3 commits June 23, 2026 21:07

DevelopmentCats changed the title ~~feat(scanner): enable SkillSpector LLM semantic pass via Coder aibridge~~ feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Claude) Jun 23, 2026

DevelopmentCats added 2 commits June 23, 2026 21:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Claude)#10

feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Claude)#10
DevelopmentCats wants to merge 13 commits into
cat/step-1-scanner-pipelinefrom
cat/scanner-llm-mode

DevelopmentCats commented Jun 22, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

DevelopmentCats commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this does

Stack

Measured impact on the five in-tree skills

Setup before merge

Graceful fallback

Why scan.yaml isn't in this PR

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DevelopmentCats commented Jun 22, 2026 •

edited

Loading